尽管基于计划的序列建模方法在连续控制方面表现出巨大的潜力，但由于高维空间中规划的高度计算复杂性和天生的困难，将它们扩展到高维状态序列仍然是一个开放的挑战。我们提出了轨迹自动编码计划器（TAP），这是一种基于计划的序列建模RL方法，可扩展到高州行动维度。使用状态条件矢量定量的变分自动编码器（VQ-VAE），点击模拟给定当前状态的轨迹的条件分布。当部署为RL代理时，TAP避免在高维连续动作空间中逐步计划，而是通过Beam Search寻找最佳的潜在代码序列。与$ o（d^3）$轨迹变压器的复杂性不同，TAP享受常数$ o（c）$规划有关州行动维度$ d $的计算复杂性。我们的经验评估还表明，随着维度的增长，TAP的表现越来越强。对于具有较高状态和动作维度的ADROIT机器人手动操纵任务，TAP超过了基于模型的方法，包括TT，其边距很大，并且还击败了强大的无模型参与者 - 批评基准。

实践和磨练技能构成了人类学习方式的基本组成部分，但很少专门培训人造代理人来执行它们。取而代之的是，它们通常是端到端训练的，希望有用的技能将被隐含地学习，以最大程度地提高某些外部奖励功能的折扣回报。在本文中，我们研究了如何将技能纳入具有较大州行动空间和稀疏奖励的复杂环境中的加固学习训练中。为此，我们创建了Skillhack，这是Nethack游戏的任务和相关技能的基准。我们评估了该基准测试的许多基准，以及我们自己的新型基于技能的方法层次启动（HKS），该方法的表现优于所有其他评估的方法。我们的实验表明，先验了解有用技能的学习可以显着改善代理在复杂问题上的表现。我们最终认为，利用预定义的技能为RL问题提供了有用的归纳偏见，尤其是那些具有较大国家行动空间和稀疏奖励的问题。

加强学习（RL）研究的进展通常是由新的，具有挑战性的环境的设计驱动的，这是一项昂贵的事业，需要技能与典型的机器学习研究人员的正交性。环境发展的复杂性仅随着程序性产生（PCG）的兴起而增加，作为产生能够测试RL剂稳健性和泛化的各种环境的流行范式。此外，现有环境通常需要复杂的构建过程，从而使重现结果变得困难。为了解决这些问题，我们介绍了基于网状引擎的基于网络的集成开发环境（IDE）Griddlyjs。 Griddlyjs允许研究人员使用方便的图形接口在视觉上设计和调试任意，复杂的PCG网格世界环境，并可视化，评估和记录训练有素的代理模型的性能。通过将RL工作流连接到由现代Web标准启用的高级功能，Griddlyjs允许发布交互式代理 - 环境演示，将实验结果直接重现为Web。为了证明Griddlyjs的多功能性，我们使用它来快速开发一个复杂的组成拼图解决环境，以及任意人为设计的环境配置及其用于自动课程学习和离线RL的解决方案。 Griddlyjs IDE是开源的，可以在\ url {https://griddly.ai}上免费获得。

事实证明，加固学习（RL）的自适应课程有效地制定了稳健的火车和测试环境之间的差异。最近，无监督的环境设计（UED）框架通用RL课程以生成整个环境的序列，从而带来了具有强大的Minimax遗憾属性的新方法。在问题上，在部分观察或随机设置中，最佳策略可能取决于预期部署设置中环境的基本真相分布，而课程学习一定会改变培训分布。我们将这种现象形式化为课程诱导的协变量转移（CICS），并描述了其在核心参数中的发生如何导致次优政策。直接从基本真相分布中采样这些参数可以避免问题，但阻碍了课程学习。我们提出了Samplr，这是一种Minimax遗憾的方法，即使由于CICS偏向基础培训数据，它也优化了基础真相函数。我们证明并验证了具有挑战性的领域，我们的方法在基础上的分布下保留了最佳性，同时促进了整个环境环境的鲁棒性。

深度加强学习概括（RL）的研究旨在产生RL算法，其政策概括为在部署时间进行新的未经调整情况，避免对其培训环境的过度接受。如果我们要在现实世界的情景中部署强化学习算法，那么解决这一点至关重要，那么环境将多样化，动态和不可预测。该调查是这个新生领域的概述。我们为讨论不同的概括问题提供统一的形式主义和术语，在以前的作品上建立不同的概括问题。我们继续对现有的基准进行分类，以及用于解决泛化问题的当前方法。最后，我们提供了对现场当前状态的关键讨论，包括未来工作的建议。在其他结论之外，我们认为，采取纯粹的程序内容生成方法，基准设计不利于泛化的进展，我们建议快速在线适应和将RL特定问题解决作为未来泛化方法的一些领域，我们推荐在UniTexplorated问题设置中构建基准测试，例如离线RL泛化和奖励函数变化。

深度强化学习（RL）的进展是通过用于培训代理商的具有挑战性的基准的可用性来驱动。但是，社区广泛采用的基准未明确设计用于评估RL方法的特定功能。虽然存在用于评估RL的特定打开问题的环境（例如探索，转移学习，无监督环境设计，甚至语言辅助RL），但一旦研究超出证明，通常难以将这些更富有，更复杂的环境 - 概念结果。我们展示了一个强大的沙箱框架，用于易于设计新颖的RL环境。 Minihack是一个停止商店，用于RL实验，环境包括从小房间到复杂的，程序生成的世界。通过利用来自Nethack的全套实体和环境动态，MiniHack是最富有的基网上的视频游戏之一，允许设计快速方便的定制RL测试台。使用这种沙箱框架，可以轻松设计新颖的环境，可以使用人类可读的描述语言或简单的Python接口来设计。除了各种RL任务和基线外，Minihack还可以包装现有的RL基准，并提供无缝添加额外复杂性的方法。

Recent progress in pretraining language models on large textual corpora led to a surge of improvements for downstream NLP tasks. Whilst learning linguistic knowledge, these models may also be storing relational knowledge present in the training data, and may be able to answer queries structured as "fillin-the-blank" cloze statements. Language models have many advantages over structured knowledge bases: they require no schema engineering, allow practitioners to query about an open class of relations, are easy to extend to more data, and require no human supervision to train. We present an in-depth analysis of the relational knowledge already present (without fine-tuning) in a wide range of state-of-theart pretrained language models. We find that (i) without fine-tuning, BERT contains relational knowledge competitive with traditional NLP methods that have some access to oracle knowledge, (ii) BERT also does remarkably well on open-domain question answering against a supervised baseline, and (iii) certain types of factual knowledge are learned much more readily than others by standard language model pretraining approaches. The surprisingly strong ability of these models to recall factual knowledge without any fine-tuning demonstrates their potential as unsupervised open-domain QA systems. The code to reproduce our analysis is available at https: //github.com/facebookresearch/LAMA.

Modeling lies at the core of both the financial and the insurance industry for a wide variety of tasks. The rise and development of machine learning and deep learning models have created many opportunities to improve our modeling toolbox. Breakthroughs in these fields often come with the requirement of large amounts of data. Such large datasets are often not publicly available in finance and insurance, mainly due to privacy and ethics concerns. This lack of data is currently one of the main hurdles in developing better models. One possible option to alleviating this issue is generative modeling. Generative models are capable of simulating fake but realistic-looking data, also referred to as synthetic data, that can be shared more freely. Generative Adversarial Networks (GANs) is such a model that increases our capacity to fit very high-dimensional distributions of data. While research on GANs is an active topic in fields like computer vision, they have found limited adoption within the human sciences, like economics and insurance. Reason for this is that in these fields, most questions are inherently about identification of causal effects, while to this day neural networks, which are at the center of the GAN framework, focus mostly on high-dimensional correlations. In this paper we study the causal preservation capabilities of GANs and whether the produced synthetic data can reliably be used to answer causal questions. This is done by performing causal analyses on the synthetic data, produced by a GAN, with increasingly more lenient assumptions. We consider the cross-sectional case, the time series case and the case with a complete structural model. It is shown that in the simple cross-sectional scenario where correlation equals causation the GAN preserves causality, but that challenges arise for more advanced analyses.

KL-regularized reinforcement learning from expert demonstrations has proved successful in improving the sample efficiency of deep reinforcement learning algorithms, allowing them to be applied to challenging physical real-world tasks. However, we show that KL-regularized reinforcement learning with behavioral reference policies derived from expert demonstrations can suffer from pathological training dynamics that can lead to slow, unstable, and suboptimal online learning. We show empirically that the pathology occurs for commonly chosen behavioral policy classes and demonstrate its impact on sample efficiency and online policy performance. Finally, we show that the pathology can be remedied by non-parametric behavioral reference policies and that this allows KL-regularized reinforcement learning to significantly outperform state-of-the-art approaches on a variety of challenging locomotion and dexterous hand manipulation tasks.

Scientists and philosophers have debated whether humans can trust advanced artificial intelligence (AI) agents to respect humanity's best interests. Yet what about the reverse? Will advanced AI agents trust humans? Gauging an AI agent's trust in humans is challenging because--absent costs for dishonesty--such agents might respond falsely about their trust in humans. Here we present a method for incentivizing machine decisions without altering an AI agent's underlying algorithms or goal orientation. In two separate experiments, we then employ this method in hundreds of trust games between an AI agent (a Large Language Model (LLM) from OpenAI) and a human experimenter (author TJ). In our first experiment, we find that the AI agent decides to trust humans at higher rates when facing actual incentives than when making hypothetical decisions. Our second experiment replicates and extends these findings by automating game play and by homogenizing question wording. We again observe higher rates of trust when the AI agent faces real incentives. Across both experiments, the AI agent's trust decisions appear unrelated to the magnitude of stakes. Furthermore, to address the possibility that the AI agent's trust decisions reflect a preference for uncertainty, the experiments include two conditions that present the AI agent with a non-social decision task that provides the opportunity to choose a certain or uncertain option; in those conditions, the AI agent consistently chooses the certain option. Our experiments suggest that one of the most advanced AI language models to date alters its social behavior in response to incentives and displays behavior consistent with trust toward a human interlocutor when incentivized.